HAND/EYE CALIBRATION METHOD USING PROJECTIVE INVARIANT SHAPE 
DESCRIPTOR OF 2 -DIMENSIONAL IMAGE 



BACKGROUND OF THE INVENTION 

5 

This application claims priority from Korean Patent Application No. 
2002-72695, filed on 21 November 2002, in the Korean Intellectual Property Office, 
the disclosure of which is incorporated herein in its entirety by reference. 

10 1. Field of the Invention 

The present invention relates to visual servoing technology, and more 
particularly, to a hand/eye calibration method using a projective invariant shape 
descriptor of a two-dimensional image. 
2. Description of the Related Art 

15 Hand/eye calibration denotes a procedure for determining the spatial 

transformation between a robot hand and a camera mounted on the robot hand to 
obtain a desired image using visual servoing technology in a robot's coordinate 
frame and control the robot. One of the hand/eye calibration methods, which is 
most frequently used, is to provide a prior motion information and to obtain desired 

20 information from images transforms generated based on the provided motion 
information. In order to easily and correctly extract transformation information of the 
robot hand, it is very important to correctly select corresponding points between 
transformed images in this method. 

25 SUMMARY OF THE INVENTION 

The present invention provides a hand/eye calibration method which makes it 
possible to easily and correctly extract transformation information between a robot 
hand and a camera. 

The present invention also provides a computer readable medium having 
30 embodied thereon a computer program for the hand/eye calibration method. 

According to an aspect of the present invention, there is provided a hand/eye 
calibration method. The method includes (a) calculating a projective invariant 
shape descriptor from at least two images consecutively obtained through a camera 
mounted on a robot hand; (b) extracting corresponding points between the images 
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by using the projective invariant shape descriptor; (c) calculating a rotation matrix for 
the corresponding points from translation of the robot; (d) calculating translation 
vector for the corresponding points from translation and rotation of the robot; and (e) 
finding a relation between the robot hand and the camera based on the rotation 
matrix calculated in step (c) and the translation vector calculated in step (d). 

According to an aspect of the present invention, there is provided a method of 
extracting corresponding points between images. The method includes (a) defining 
errors for a projective invariant shape descriptor for a two-dimensional image from at 
least two images obtained at a predetermined interval and calculating noisy 
invariance; (b) calculating a threshold to be used to set corresponding points 
according to the noisy invariance; (c) extracting boundary data from the images and 
presenting the extracted boundary data by subsampling N data; (d) minimizing the 
projective invariant shape descriptor; (e) transforming a following image into the 
following image according to the minimized projective invariant shape descriptor; (f) 
resetting distance between boundary data in consideration of the ratio of distance 
between boundary data before the transformation to distance between boundary 
data after the transformation; and (g) finding similarities between the boundary data 
and extracting corresponding points between the previous images and the following 
image. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The above and other features and advantages of the present invention will 
become more apparent by describing in detail exemplary embodiments thereof with 
reference to the attached drawings in which: 

FIG. 1 is a view of a robot hand on which a camera is mounted, the robot 
hand being controlled using visual servoing; 

FIG. 2 is a view of an appearance of the camera of FIG. 1 ; 

FIG. 3 is a view of a pin-hall camera model of the camera of FIG. 2; 

FIG. 4 is a view of coordinate systems of an object plane and an image plane, 
and a projection result of the image plane to the object plane; 

FIG. 5 is a view showing conditions to satisfy a linearity of the camera of FIG. 

2; 



FIG. 6 is a view presenting a relationship for a coordinate system: of an object 
plane, a coordinate system on an image plane, and a world coordinate system of a 
robot hand; 

FIG. 7 is a view for explaining a method of obtaining a projective invariant 
5 shape descriptor from a 2-dimensional image;. 

FIG. 8 is a view showing an example of projective invariant shape descriptor 
calculated from a 2-dimensional shape of FIG. 7; 

FIG. 9 is a flowchart of a method for hand/eye calibration according to an 
embodiment of the present invention; and 
10 FIG. 10 is a detailed flowchart of a method of obtaining corresponding points 

between a previous image and a following image shown in FIG. 9. 

DETAILED DESCRIPTION OF THE INVENTION 
The present invention will now be described more fully with reference to the 
15 accompanying drawings, in which preferred embodiments of the invention are shown. 
In the drawings, like reference numerals refer to like elements throughout. 

FIG. 1 is a view of a robot hand 100 on which a charged-coupled device 
(CCD) camera 110 is mounted to adopt a visual servoing controlling technique. 
Referring to FIG.- 1 , at least one CCD camera 1 10 is mounted at an end of the robot 
20 hand 100 used as an industrial robot. 

FIG. 2 is a view of an appearance of the CCD camera 110 of FIG. 1. 
Referring to FIG. 2, the CCD camera 110 is composed of a main body 111, a CCD 
array 112, and a lens 114. The lens 114 has the same function as that of a 
crystalline lens of a person's eyes. The CCD array 112 corresponds to an image 
25 plane on which an image projected through the lens 1 14 of the CCD camera 1 10 is 
cast and has the same function as that of the retina of a person's eyes. When a 
focus of the lens 1 14 is set to infinitely, the length from a center of the lens 1 14 to the 
CCD array 112 denotes a focal length F, and the image looks different according to 
the focal length F. The focal length F is an important parameter necessary to 
30 estimate the length between the CCD camera 110 and an object. 

FIG. 3 is a view of a pin-hall camera model of the CCD camera 1 10 of FIG. 2, 
and FIG. 4 is a view of coordinate systems of an object plane and an image plane, 
and a projection result of the image plane to the object plane. Referring to FIGS. 3 
and 4, a projective transformation for an image in the pin-hall camera model can be 
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expressed as follows. 
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where (u, v, 1 ) denotes a coordinates of a point q defined on the image plane, 
(X, Y, Z, 1) denotes a coordinates of a point P in an object coordinate system, and 
^ denotes an ij factor of a transformation matrix between an object plane and the 
image plane. 

Here, if an object is projected to on a two-dimensional plane, i.e., Z=0, 
Equitation 1 is transformed as follows. 
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10 As shown in Equations 1 and 2, the process for obtaining an image is 

performed in non-linear environment. However, a linearized projective 
transformation is adopted to a two-dimensional image obtained through the CCD 
camera 110 rather than a non-linear projective transformation like in Equation 2. 

FIG. 5 is a view showing conditions to obtain a linear model of the CCD 

15 camera 110 of FIG. 2. As shown in FIG. 5, if a length S from the CCD camera 110 
to the object is sufficiently longer than a size S 0 of the object, the non-linear 
projective transformation from like Equation 2 is transformed as follows. 

"1 tn t X2 tl4 1 \X~ 

v ttx t 22 Y. (3) 

!J ^2 /34j L 1 

A Fourier descriptor is a linearized shape descriptor which satisfies 
20 Equations 1 , 2, and 3. The Fourier descriptor represents an image of the object 
with Fourier coefficients which are obtained by a two-dimensional Fourier 
transformation for the image contour of a two-dimensional object. However, this 
method can be applied only to a case where linearity of a camera is guaranteed, that 
is, where a distance between the CCD camera 110 and the object is too long. 
25 Therefore, to overcome the restriction, the image obtained from the CCD camera 
110 is analyzed by using a projective invariant shape descriptor I in the present 
invention. As a result, even in a case where the linearity of the camera is not 
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guaranteed, that is, the distance between the CCD camera 110 and the object is not 
long, the image can be analyzed correctly without being affected by noise, slant 
angles, or the nonlinearity of the CCD camera 110 occurring when images are 
obtained. 

5 FIG. 6 is a view presenting a relationship for a coordinate system of an object 

plane, a coordinate system on an image plane, and a world coordinate system of a 
robot hand. Referring to FIG. 6, a coordinate system of the CCD camera 110 
corresponds to a world coordinate system of the robot hand 100 after a rotation and 
a translation. Thus, a robot hand/eye calibration is a process of finding the 
10 elements, a rotation matrix R, which gives a direction of the CCD camera 110, and 
components of a translation vector t, which gives a location of the CCD camera 110, 
in the world coordinate system. 

In the hand/eye calibration method according to the present invention, the 
pin-hall camera model is used in which distortion of the lens 114 or misalignment of 
15 a light axis can be ignored. The relations between the robot hand 100 of FIG. 6 and 
the CCD camera 110 can be expressed as follows, 

X h = RX c + t (4) 

where x h denotes the world coordinate system of the robot hand 100, i.e., 
X c denotes a coordinate system of the CCD camera 110, R denotes a rotation 
20 matrix, and t denotes a translation vector. 

The relation between the CCD camera 110 and the image can be expressed 
. as follows. 

(h-ho) = £— , (v-v 0 ) = ^^ ,.. - (5). 

S'x Zc Sy Zc 

where, wand Uo denote X coordinates on an image plane , and v and Vo 
25 denote Y coordinates on an image plane. In addition, / denotes a focal length 
between the lens 114 and the CCD array 112, and S x and Sy denote scale factors of 
the CCD camera 110. The focal length / and scale factors S x and S y are 
characteristic values which indicate original characteristics of the CCD camera 110 
and they are fixed according to a specification of the CCD camera 110. 
30 If robot motion information already known to the user x P i = R P ,X p2 + t pl is 

introduced in Equation 4, the following Equation is obtained. 

RX cl + t = R pl {RXci + t) + tpl (6) 
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The motion of the CCD camera 110 x c \ can be expressed as follows by 
using Equation 6. 



■(7) 
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X c \— Xc2^ 
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If the rotation matrix is excluded from Equation 7 ;and only translation is 
considered, the rotation matrix R can be expressed as follows. 



td^R t 



(8) 



Equation 8 can be expressed by substituting t pi with three motion vectors of 
the robot, t pi , t p2l and t p3 as follows. 



5 ^ c <2 5 tc'3y~ R l (tn\ , t p2 > t pi) 



(9) 



10 Here, image vectors corresponding to three motion vectors of the robot hand, 

t P i, t p2i and t p3 are OFi, OF 2 , and OF 3 , and each image vector is defined by the 
following Equation 10, 
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OFi = f 



(Uj ~ Mq) (y/ - Vp) 
fx ' /, 
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where f x = — , and f y = -*—. Intrinsic parameters can be calculated as 

Sx S y 



follows. 



[OFx-OF 2 = 0] 

— t(ui ~ Uo)(u 2 - wo) + ~ v 0 )(v 2 - v 0 ) + 1 = 0 

J X J y 

[OFrOFi = 0] 

~l \U\ - W 0 )(w 3 ~ Wo) + ~T (Vi - V 0 )(V 3 - V 0 ) + 1 = 0 
J x J y 

[OF 2 OF 3 = 0] 

—1~(U2 - Wo)(«3 - Wo) + ~~j(v 2 - V 0 )(v 3 - V 0 ) + 1 = 0 

J X J y 

Equation 1 1 can be expressed as follows, 

Wo (w 2 ~ W 3 ) + Si (V 2 - V 3 ) - S 2 Vi (v 2 - V 3 ) = U\ (w 2 ~ W 3 ) 
Wo (wi - W 3 ) + Si (vi - V 3 ) - S2 v 2 (vi - V 3 ) = w 2 (wi - W 3 ) 
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where c. = 



r, 

S\ v 0 — 7, 

/ y 
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and S2 = ^. Equation 12 can be expressed in a matrix 

f y 



form as follows. 
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(1/2-W3) (v 2 -v 3 ) -Vi(v 2 -v 3 ) 

(Wi - W 3 ) (Vi - V 3 ) - V 2 (V| - V 3 ) 

{u\ - w' 3 ) (v' 2 - v' 3 ) - v', (v' 2 - v' 3 ) 
(vW 3 ) ~v\{v\-v\) 

In consideration of rotation and translation of the robot hand 100, the 
translation vector t between the robot hand 100 and the CCD camera 110 can be 
expressed as follows, 

t c \ = R 1 (R P i* + t p i - 0 (14) 

t = {Rpi~ iy x {Rt c i-t P d .......................................... 

where (R ph t pi ) denotes motion information already known by the user, and R 

denotes a rotation matrix which is calculated from three rotations of the robot hand 

100. t c -i denotes an image vector, and / denotes a projective invariant shape 

descriptor calculated from a two-dimensional image. In order to improve the 

precision of the hand/eye calibration, it is very important to correctly set points 

corresponding to coordinates which are predetermined within the field of view of the 

CCD camera 110. Therefore, in the present invention, corresponding points are 

obtained by the CCD camera 110 by using the projective invariant shape descriptor 

which does not vary under nonlinear transformation. Then, the corresponding 

points are used as calibration targets to conduct hand/eye calibration. The 

projective invariant shape descriptor /, which is used as a fundamental factor of the 

hand/eye calibration, can be defined as follows. 

j = det (g 5 <h jA dct( ?5 <?2 9 3 ) = det(p 5 p { p 4 ) detCP 5 p 2 p 2 ) 
fet(q 5 q x q z )&zt(q.q 2 q 4 ) det(p 5 p { p 3 ) det(p 5 p 2 p 4 ) 

where P denotes points of the object, q denotes corresponding points of the 
image as shown in FIG. 3. det(- ) in Equation 15 can be defined as follows. 



det(^^ 2 ^ 3 ) = / 



det(p,/> 2 p 3 ) = / 
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= 2" (Area of Ap,p,? 3 ) 
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The projective invariant shape descriptor / expressed in Equations 15 and 16 
represents information which does not vary under nonlinear transformation as shown 
in Equation 2 and does not vary though images obtained by the CCD camera 110 
are transformed. 

5 FIG. 7 is a view for explaining a method of obtaining a projective invariant 

shape descriptor from a 2-dimensional image. Referring to FIG. 7, a contour of the 
two-dimensional image of a drawing is extracted, and the extracted contour is 
divided into five similar intervals. Coordinates of points (Xi(1),' X-i(k), X 2 (1), X 2 (k), 
X 3 (1), X 3 (k), X 4 (1), X4(k) f X 5 (1), X 5 (k)) which constitute each interval are obtained, 
10 and then the projective invariant shape descriptor is calculated. The points (X^l), 
Xi(k), X 2 (1), X 2 (k), X 3 (1), X 3 (k), X4(1), X4(k), X 5 (1), X 5 (k)) are input into Equation 15 
recursively, with moving continuously by 1/N times the length of the contour along 
the contour until each point reaches its initial location. 

15 

where x,(k) = (X(k), Y(k),l), 

x 2 (k) = (X(j+k),r(j+k),i), 

XAk) = (X(j + k),Y(j + k),l), 

2N 2N 
Xi (k) = (X(— + k),Y(^ + k),l), 

3N 3N 

20 X4 (k) = (X(-j- + k), y(.~- + k), 1) , 

AN AN 

Xs (k) = {X(_— + k),Y{—-+k),\), and \<k<N , and X(k) and Y(k) denotes X 

and Y axis coordinate function of the contour. 

An example of projective invariant shape descriptor calculated from a 
2-dimensional shape of FIG. 7 is shown in FIG. 8. The calculation result of the 
25 projective invariant shape descriptor / of FIG. 8, i.e., projective invariant, maintains 
its intrinsic value though the shape of the image obtained by the CCD camera 110 is 
transformed. In addition, the projective invariant shape descriptor / is not affected 
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by noise or slant angles. Thus, if the projective invariant is used for the hand/eye 
calibration, precision of the calibration can be improved. 

Corresponding points of the images obtained by the CCD camera 110 by 
using the projective invariant shape descriptor / can be extracted as follows. 
5 In order to extract corresponding points of the images, errors in the projective 

invariant shape descriptor / have to be defined. In the present invention, the errors 
are defined using a Gaussian noise model. In order to use the Gaussian noise 
model, Equation 17 can be expressed as follows. 

, / = det(X 5 XiXddet(X 5X2X3) (18) 
det(X 5 X\ X3) det(^ 5 X 2 X 4 ) 

10 where x, = U,y i -,i) r or I = l0cuy x ,x 2 ,y 29 x3,y 3 ,x4,y 4 ,x5,y 5 )> 

Here, if (x it y t ) is true data, and (x,,?, ) is a noisy observation parameter, the 
noise observation parameter can be expressed as follows, 

x/ = *i + £p Ji = yi + Vi (19) 

where noise terms and 7/ are distributed noise terms, and their mean and 

15 variance are 0 and <?] . The noise terms can be expressed as follows. 

|o otherwise\ ' ^ ,7 ^ |q otherwise^ ^ ^ 

= 0 

20 Noisy invariant can be expressed as follows after noisy measurements on the 

image are observed. 

7(x,J l} x 2 J 2 ,x 3 J 3 ,x 4i y 4y x 5 ,y 5 ) (21) 

In order to calculate an expected value and a variance of the noisy invariant 

/ , the noisy invariant / can be expressed with (x It y>, x 2 , yi> x 3> y 3 , x 4i y 4l x 5 , y 5 ) by 
25 using Talyor series. 

7 « / + Z [(*, - x t ) |4 + (5> ■ - y t ) |4] 
m dx ( dy. t 

= / + iK,.|4+7,|^] , (22) 

,=i 1 d x { 1 d y t 

Here, the variance can be expressed as follows. 

— — 2 

^[(/-/) 2 ] = cr^S[(|4) 2 + (|4) ] (23) 
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A threshold of the noisy invariant can be defined as follows. 

A/ = 3xV£[(7-/) 2 ] 



(24) 



The corresponding points are found from the images obtained by the CCD 
camera 110 of the robot hand 100 by repeating calculation of the projective invariant 
shape descriptor, and boundary data between a previous image and a following 
image consecutively obtained through the CCD camera 110 can be expressed as 
follows. 

Oi n = {X%,Y%},k = l~n !n 

0?°={X%°,Y% 0 },k = l~n Mo 

where n In and n Mo denote the number of points in a boundary between a 
scene and a model. The boundary data are presented by subsampling N data, and 
this subsampling can be expressed as follows. 



(25) 



(26) 



n In Mo 

where ri (/) = — x/, T2 (/) = - — xz, and N denotes the number of points on a 

N N 

normalized contour. 

Then, a projective invariant shape descriptor is calculated by using qf and 
qf° defined in Equation 26 when a value of the following Equation is minimized, 

e 1 = £ wj (qf - (C • qf° + d)'' (A qf° + b)f (qf - (C ■ qf° + d)" (A qf° + b)) . . : . . . (27) 

where A, b, c, and d denote variants defined from transformation between the 
previous image and the following image and can be expressed as follows. 
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(28) 
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The weight w,- of Equation 27 is calculated by the variance defined in Equation 
23 as follows. 
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The projective invariant shape descriptors can be minimized by Equations 27 
through 30 as follows. 

P = {Q T QY X Q T H . 

Here, P={s,, s ' 2 , s ' 4 , s 5 , s ' 3 , s ' 6 , s V, s ' 8 ) T , 



(31) 



Q = 



0 0 0 
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After the projective invariant shape descriptors are minimized by using 
Equation 31, the following image obtained by the CCD camera 110 is transformed 
into a previous image. This transformation can be expressed as follows. 

qT={C.qT + dy\AqT + b\ i = l~N (32) 

where A, b, c, and d denotes variance defined transformation between the 
previous image and the following image. 

After the transformation between the previous image and the following image 
is completed, the ratio of the distance between boundary data before the 
transformation to the distance between boundary data after the transformation, i.e., 
r 2 (0'. is calculated by using the following Equation, and then the length between 
data is reset by using the ratio. 



r\ (0 



Mo' Mo'\ 

\<1m ~1i 



i = 1 ~ N 



= 2L\q M ~q t 



Mo'\ 



(33) 



20 By using the ratio r 2 (0' calculated by Equation 33, the following image can 

be resampled as follows. 

<j f "' = O?2 (0 , i = l~~N : (34) 

After that, in order to include errors between the previous image and the 
following image within a predetermined scope, Equations 29 through 34 are 
25 repeated. The errors are expressed by errors of the corresponding points and 
similarities between the projective invariant shape descriptors (I m , /,) of the boundary 
data. A similarity value of the projective invariant shape descriptors (I m , Ig) of the 
boundary data can be expresses as follows. 
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similarity{l m ,7, ) = 



N 




1, //KW-/,(A:)|<A/ 



0, otherwise 



(35) 



If the maximum value of the similarity values is greater than a predetermined 
threshold, it is determined that the corresponding points of the previous image and 
the following image are the same as each other. The value of AI in Equation 35 
and the predetermined threshold are selected according to an environment to which 
the present invention is applied and a required precision. 

FIG. 9 is a flowchart of a method for hand/eye calibration according to a 
preferred embodiment of the present invention. Referring to FIG. 9, the hand/eye 
calibration method according to the present invention includes calculating a 
projective invariant shape descriptor / from a two-dimensional image (step 210). 
Then, corresponding points between a previous image and a following image 
consecutively obtained through the CCD camera 110 are extracted by using the 
calculated projective invariant shape descriptor / (step 220). The extracted 
corresponding points are used as targets to perform the hand/eye calibration. 

Then, a rotation matrix R for a coordinate, i.e., the extracting corresponding 
points, is calculated from a translation of the robot hand 100 (step 230). Here, the 
rotation matrix R is calculated by Equation 8. A translation vector t for the 
coordinate, i.e., the extracted corresponding points, is calculated from translation 
and rotation of the robot hand 100 (step 240). Here, the translation vector t is 
calculated by Equation 14. After completion of steps 230 and 240, the hand/eye 
calibration which defines a relation between the robot hand 100 and the CCD 
camera 110, that is, obtains a calculation result of X h = RX c +t , is completed (step 
250). Here, X h denotes a coordinate system of the robot hand 100, and X c denotes 
a coordinate system of the CCD camera 110 mounted on the robot hand 100. 

FIG. 10 is a detailed flowchart of a method of obtaining corresponding points 
between a previous image and a following image shown in FIG. 9. In general, a 
calibration target which is finely organized is required to accurately perform the 
hand/eye calibration. Thus, as shown in FIG. 10 of the present invention, 
corresponding points of consecutive images obtained by the CCD camera 110 are 
extracted by using the projective invariant shape descriptor /which is not affected by 
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nonlinearity or noise of the CCD camera 110, and the projective invariant shape 
descriptor / is used as the calibration target. 

In order to accurately perform the hand/eye calibration, errors of the projective 
invariant shape descriptor / of images are defined (step 2200), and noisy invariant is 
calculated by analyzing an amount of noises of the images (step 2210). Then, a 
threshold is calculated according to the noisy invariant calculated in step 2210 (step 
2220). 

Boundary data of a previous image and a following image obtained by the 
CCD camera 110 are extracted (step 2230). The extracted boundary data is 
presented by subsampling N data (step 2240). Then, a projective invariant shape - 
descriptor is minimized in accordance with Equation 31 (step 2250), and the 
following image is transformed into a previous image in response to the minimized 
projective invariant shape descriptor (step 2260). After that, the distance between 
boundary data is reset by using the ratio of the distance between boundary data 
before the transformation to the distance between boundary data after the 
transformation (step 2270). 

After the distance between the boundary data is reset in step 2270, similarities 
between the boundary data of the previous image and the following image are 
founded (step 2280), and corresponding points between the previous image and the 
following image are extracted by using the found similarities (step 2290). 

The hand/eye calibration method according to the present invention extracts 
corresponding points between the previous image and the following image by using 
the projective invariant shape descriptor/. Thus, it is possible to accurately perform 
the hand/eye calibration without being affected by noise or nonlinearity of a camera. 

In the present invention, the hand/eye calibration method is applied to the 
robot hand. However, the hand/eye calibration method can be applied to various 
kinds of visual servoing devices which control motions of an object by using images 
obtained through at least one camera. 

The present invention may be embodied as a computer readable code in a 
computer readable medium. The computer readable medium includes all kinds of 
recording device in which computer readable data are stored. The computer 
readable medium includes, but not limited to, ROM's, RAM's, CD-ROMs, magnetic 
tapes, floppy disks, optical data storage device, and carrier waves such as 
transmissions over the Internet. In addition, the computer readable medium may be 
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distributed to computer systems which are connected via a network, be stored and 
embodied as the computer readable code. 

While the present invention has been particularly shown and described with 
reference to exemplary embodiments thereof, it will be understood by those of 
5 ordinary skill in the art that various transforms in form and details may be made 
therein without departing from the spirit and scope of the present invention as 
defined by the following claims. 
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